{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Splitters\n",
    "Once you've loaded documents, you'll often want to transform them to better suit your application. The simplest example is you may want to split a long document into smaller chunks that can fit into your model's context window. LangChain has a number of built-in document transformers that make it easy to split, combine, filter, and otherwise manipulate documents.\n",
    "\n",
    "加载文档后，您通常需要转换它们以更好地适应您的应用程序。最简单的示例是，您可能希望将长文档拆分为更小的块，以便适合模型的上下文窗口。LangChain有许多内置的文档转换器，可以很容易地拆分、组合、过滤和以其他方式操作文档。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Types of Text Splitters\n",
    "LangChain offers many different types of text splitters. These all live in the langchain-text-splitters package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Name | Classes | Splits On | Adds Metadata | Description |\n",
    "| --- | --- | --- | --- | --- |\n",
    "| Recursive | [RecursiveCharacterTextSplitter](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/recursive_text_splitter/), [RecursiveJsonSplitter](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/recursive_json_splitter/) | A list of user defined characters |  | Recursively splits text. This splitting is trying to keep related pieces of text next to each other. This is the `recommended way` to start splitting text. |\n",
    "| HTML | [HTMLHeaderTextSplitter](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/HTML_header_metadata/), [HTMLSectionSplitter](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/HTML_section_aware_splitter/) | HTML specific characters | ✅ | Splits text based on HTML-specific characters. Notably, this adds in relevant information about where that chunk came from (based on the HTML) |\n",
    "| Markdown | [MarkdownHeaderTextSplitter](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/markdown_header_metadata/) | Markdown specific characters | ✅ | Splits text based on Markdown-specific characters. Notably, this adds in relevant information about where that chunk came from (based on the Markdown) |\n",
    "| Code | [many languages](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/code_splitter/) | Code (Python, JS) specific characters |  | Splits text based on characters specific to coding languages. 15 different languages are available to choose from. |\n",
    "| Token | [many classes](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/split_by_token/) | Tokens |  | Splits text on tokens. There exist a few different ways to measure tokens. |\n",
    "| Character | [CharacterTextSplitter](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/character_text_splitter/) | A user defined character |  | Splits text based on a user defined character. One of the simpler methods. |\n",
    "| \\[Experimental\\] Semantic Chunker | [SemanticChunker](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/semantic-chunker/) | Sentences |  | First splits on sentences. Then combines ones next to each other if they are semantically similar enough. Taken from [Greg Kamradt](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb) |\n",
    "| AI21 Semantic Text Splitter | [AI21SemanticTextSplitter](https://python.langchain.com/v0.1/docs/integrations/document_transformers/ai21_semantic_text_splitter/) | ✅ | Identifies distinct topics that form coherent pieces of text and splits along those. |  |"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain0_1",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
